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EXECUTIVE SUMMARY: 
The Impacts of i-Ready Personalized Instruction on Student Math 
Achievement in Multiple School Districts 


In July 2021, The Center for Research and Reform in Education (CRRE) at Johns 
Hopkins University contracted with Curriculum Associates (CA) to conduct a quantitative 
efficacy study of the effects of i-Ready Personalized Instruction on student achievement in five 
Massachusetts school districts. The present report examines findings from quantitative analyses 
comparing achievement gains, as measured by the Massachusetts Comprehensive Assessment 
System (MCAS), between students who experienced both i-Ready Personalized Instruction and 
Diagnostic testing and students who only participated in i-Ready Diagnostic testing. Additional 
analyses examined relationships between i-Ready usage metrics and both i-Ready and MCAS 
scores. Outcome data from the 2020-21 school year, which was disrupted due to the COVID-19 
pandemic, were used in all present analyses. 


Research questions for this evaluation include the following: 


1. What is the effectiveness of i-Ready Personalized Instruction on student achievement on 
summative state assessments in mathematics in a year of learning disruptions from the 
COVID-19 pandemic? 

2. How are the effects of i-Ready Personalized Instruction on achievement impacted by 
student and implementation variables? 

a. By student prior achievement demographic characteristics (subgroups), such as 
grade level 
b. Student i-Ready testing and instruction usage? 


The study sample consisted of about 11,000 students in grades 3-8 from five school 
districts in Massachusetts. All schools used i-Ready Diagnostic assessments, but each district 
assigned some students to receive both the Diagnostic and Instruction products, while other 
students only received the Diagnostic product. 


Key findings of the current study include: 


i-Ready Personalized Instruction was associated with mathematics achievement 


gains. Use of i-Ready Personalized Instruction was associated with significantly higher 
mathematics achievement on the MCAS in grades 3, 5, and 6. i-Ready Personalized Instruction 
students scored 5-7 points higher, on average, than did comparison students who only 
participated in the i-Ready Diagnostic Assessment program. In grade-aggregated analyses, i- 
Ready Personalized Instruction students showed significantly higher mathematics achievement 
than did comparison students, with Instruction students averaging five points higher MCAS 
scores. 
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i-Ready Personalized Instruction usage metrics were higher in elementary schools. 
Instructional usage metrics were considerably higher in elementary school grades than in middle 
school grades. Specifically, elementary students averaged four more hours of usage and 16 more 
completed lessons than did middle school students. 


i-Ready Personalized Instruction usage that met usage guidance was associated 
with mathematics gains. Students who met Curriculum Associates’ recommended i-Ready 
usage guidelines of 30 minutes per week for at least 18 weeks and passing more than 70% of 
lessons had significantly higher mathematics achievement in relation to comparison students. 
Specifically, mathematics Instruction students who met Curriculum Associates’ usage guidelines 
had 6—17-point higher MCAS math scores than did comparison students. 


Conclusions 
The key results and conclusions of this evaluation are as follows: 


e Students in grades 3, 5, and 6 who used i-Ready Personalized Instruction showed 
significantly higher mathematics achievement on the MCAS mathematics assessment 
than did comparison students. 

e i-Ready Personalized Instruction metrics including total time and lesson count variables 
were significantly positively associated with MCAS mathematics achievement across all 
grades. 

e i-Ready Personalized Instruction usage was generally higher in elementary grades, in 
relation to middle school grades. 

e All quartiles of i-Ready Personalized Instruction usage were associated with significantly 
higher mathematics achievement, in relation to comparison students, in grades 5 and 6. 
Quartiles 2-4 of i-Ready Personalized Instruction usage were associated with 
significantly higher mathematics achievement in grade 3, as well. 

e Students who met i-Ready Personalized Instruction usage guidelines generally showed 
significantly higher MCAS mathematics achievement, in relation to Diagnostic-only 
students. 
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The Impacts of i-Ready Personalized Instruction on Student Math 
Achievement in Multiple School Districts 


In July 2021, The Center for Research and Reform in Education (CRRE) at Johns 
Hopkins University contracted with Curriculum Associates (CA) to conduct a quantitative 
efficacy study of the effects of i-Ready Personalized Instruction on student achievement in five 
Massachusetts school districts. The present report examines findings from quantitative analyses 
comparing achievement gains, as measured by the Massachusetts Comprehensive Assessment 
System (MCAS), between students who experienced i-Ready Personalized Instruction and 
Diagnostic testing and students who only participated in Diagnostic testing. Additional analyses 
examined relationships between i-Ready usage metrics and both i-Ready and MCAS scores. 
Outcome data from the 2020-21 school year, which was disrupted due to the COVID-19 
pandemic, were used in all present analyses. 


The i-Ready Diagnostic assessment is an adaptive assessment designed to provide 
teachers with actionable insight into student needs. The Diagnostic assessment offers a complete 
picture of student performance and growth, eliminating the need for multiple, redundant tests. 
The i-Ready Diagnostic assessment pinpoints student ability level, identifies specific skills 
students need to learn to accelerate their growth, and charts a personalized learning path for each 
student. 


The i-Ready Personalized Instruction suite delivers online lessons for grades K-8 students 
that provide tailored instruction that meets learners at their level, helps them problem solve, and 
keeps students motivated to continue their progress. Instruction uses data obtained from the i- 
Ready Diagnostic assessment to deliver personalized learning paths for each student, balancing 
rigor and reachability. Online lessons offer students explicit instruction when they need it, along 
with systematic practice and scaffolded feedback that helps to promote a growth mindset. 


Research questions for this evaluation include the following: 


1. What is the effectiveness of i-Ready Personalized Instruction on student achievement on 
summative state assessments in mathematics in a year of learning disruptions from the 
COVID-19 pandemic? 


2. How are the effects of i-Ready Personalized Instruction on achievement impacted by 
student characteristics and implementation variables? 
a. By student prior achievement demographic characteristics (subgroups), such as 
grade level 
b. Student i-Ready testing and instruction usage? 


Method 


Research Design 
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This study was a quasi-experimental design (QED) that analyzed end-of-year summative 
state test data and i-Ready Diagnostic assessment and usage data from the 2020-21 school year. 
Specifically, Mathematics MCAS scores from the 2020-21 school year were obtained for all 
students in grades 3-8. Mathematics MCAS scores were also obtained from the 2019-20 school 
year, but due to sparseness of data (only about 10% of observations contained non-missing data), 
MCAS scores from this school year were not used in analyses. We also obtained i-Ready 
Diagnostic scores from the fall, winter, and spring of the 2020-21 school year, along with i- 
Ready usage data for students who used i-Ready Personalized Instruction. Fall 2020 i-Ready 
scores were used as the prior achievement variable, in place of spring 2020 MCAS scores. As i- 
Ready Personalized Instruction usage tends to be implemented by school, Hierarchical Linear 
Modeling (HLM) was used to compare student achievement between students who did and did 
not receive i-Ready Personalized Instruction. We also conducted similar analyses to examine the 
relationships between usage variables and ELA and mathematics achievement. 


Participants 


Student data were originally obtained from a total of just over 18,000 students from five 
school districts in Massachusetts. We received data for all grades K-8 students in these five 
districts, but since only grades 3-8 students had outcome (MCAS) variable data, we dropped 
grades K-2 students from our analytic sample, leaving a sample of approximately 11,000 
students from 69 schools. 


Student demographics for participants in this study are displayed in Table 1. “Other 
Race” is defined as ethnicities other than White, Hispanic, or Black. The treatment sample 
contained significantly higher percentages of Hispanic, Black, and ELL students, while the 
comparison sample contained a larger percentage of non-Hispanic White students. 


Table 1 
Student characteristics for analytic sample, by percentage 
Treatment Comparison 

% White 64.52 75.86* 
% Hispanic 64.52* 12.83 
% Black 27.66* 13.32 
% Other Race 3.58 7.50 
% Female 49.33 49.83 
% Students with Disabilities/SPED 25.43 19.12 
% ELs 13.97* 3.10 
N 7,646 3,881 

Note: * p < .05 

Measures 


Data sources for the current study include student i-Ready Diagnostic scores, i-Ready 
Personalized Instruction usage data, student demographic data, and student MCAS achievement 
data. Mathematics scores were obtained from both i-Ready and MCAS assessments. Student 
achievement data from the 2020-21 school year were analyzed to compare achievement gains 
between students who did and did not receive i-Ready Personalized Instruction throughout the 
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school year. In addition, i-Ready Personalized Instruction usage data were analyzed to examine 
relationships between i-Ready usage and MCAS test scores, in relation to comparison students 
who did not receive i-Ready Personalized Instruction. 


MCAS scores. MCAS mathematics scores were obtained from the spring of the 2019-20 
and 2020-21 school years for all grades 3-8 students. Initially, spring 2020 scores were intended 
to be used as prior achievement controls for our analytic models; however, the sparseness of the 
data (only about 10% of students recorded valid scores for this administration), due to the 
COVID-19 pandemic, rendered this not feasible, and were replaced in our models by fall 2020 i- 
Ready Diagnostic scores. Spring 2021 mathematics scores were used as the outcome variable in 
our analyses. MCAS scores ranged from 440-560 and are not vertically scaled, meaning a score 
of 500 in grade 4 is not equivalent to a score of 500 in grade 5 in terms of academic 
achievement, for example. Table 2 shows the classification of MCAS scores into achievement 
levels across all grades and subjects. 


Table 2 

MCAS achievement level score bands 
Achievement Level Scaled Scores 
Not Meeting Expectations 440-469 
Partially Meeting Expectations 470-499 
Meeting Expectations 500-529 
Exceeding Expectations 530-560 


Demographic variables. The data also included a series of demographic variables 
including race, gender, ethnicity, economic disadvantage, special education, and English 
Language Learner status. Not all the districts included data on all these variables, especially 
economic disadvantage and ELL status. 


i-Ready Diagnostic Scores. Overall and sub-domain i-Ready Diagnostic assessment 
scores were obtained for all elementary and middle school students (grades K-8) in the 2020-21 
school year. Mathematics sub-domains included numbers and operations, algebra, geometry, and 
measurement. We focused on overall mathematics scores for the present analyses. We obtained 
fall, winter, and spring i-Ready Diagnostic scores, but focused mainly on the fall scores as a 
prior achievement control in our main achievement analyses. i-Ready Diagnostic assessment 
scores range from 0-800 and are vertically scaled and nationally normed across grades, meaning 
that scores can be directly compared to each other, regardless of a student’s current grade level. 
In our analyses, i-Ready Diagnostic scores tended to range between 400-700. 


i-Ready Usage data. i-Ready mathematics usage data were obtained for all students who 
were tested by i-Ready in the 2021 school year. The usage data consists of time spent on lessons 
and instruction only and does not include time spent on diagnostic assessments. Thus, students 
who were Diagnostic-only (comparison students) had 0’s on nearly all usage metrics. Usage 
metrics included: total lessons completed, unique lessons completed, passed lessons, minutes of 
usage, weeks of instruction, and weeks with at least one completed lesson. We focused on total 
instructional time, lessons completed, unique lessons completed, and passed lessons in our main 
analyses. 
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Analytical Approach 


Data for students in grades 3-8 were analyzed by descriptively examining patterns of 
MCAS and i-Ready Diagnostic scores and usage, as well as by comparing achievement patterns 
between students who received i-Ready Personalized Instruction (Treatment students) and 
students who only received i-Ready Diagnostic assessments (Comparison students). Hierarchical 
Linear Modeling (HLM) at each grade level was used to compare differences in achievement, as 
measured by the MCAS, between treatment and comparison students. Schools were used as the 
Level 2 (cluster-level) variable, as i-Ready Personalized Instruction usage is typically clustered 
at the school level. Demographic variables such as gender, ethnicity, ELL status, and special 
education status were included in all models. We also included HLM models that tested the 
unique effect of each usage metric by replacing the treatment variable (i-Ready Personalized 
Instruction vs. Diagnostic testing only) with the usage metric. This allowed us to estimate the 
effect of individual units of instruction, such as one hour of usage or one completed lesson, on 
MCAS scores, in relation to students who did not use i-Ready Personalized Instruction. All 
covariates in regression models were grand mean centered to enable interpretation of the 
intercept. 


Initially, baseline equivalence was not met for fall 2020 mathematics i-Ready scores, 
across all grade levels. Baseline equivalence is defined as being met if the standardized mean 
difference between treatment and comparison groups is less than 0.25 SD (WWC, 2020). Here, 
the differences all favored the Comparison group, and ranged between approximately 0.60 and 
0.87 SDs. Unadjusted means for 2020 mathematics i-Ready scores by grade are presented in 
Table 3. 


Table 3 
Baseline equivalence, unadjusted, by grade 
Grade Treatment Comparison Stan. Mean Diff. 
Grade 3 416.14 438.18 -0.873 
Grade 4 434.50 452.71 -0.724 
Grade 5 452.01 469.54 -0.662 
Grade 6 465.02 482.63 -0.606 
Grade 7 474.39 495.62 -0.716 
Grade 8 483.00 506.98 -0.664 


Note: Fall 2020 i-Ready is baseline achievement variable 


To adjust for the large standardized mean differences between treatment and comparison 
students on baseline achievement, propensity score weighting (PSW) was used in all analyses for 
the purpose of creating comparison groups that were as similar as possible to groups of treatment 
students. As analyses were intended to be performed by grade-level, PSW was also conducted 
separately at each grade level. Within each grade level, treatment students were each given a 
weight of one, and comparison students were each given a weight of: 


Probability; 


Weight; = 1 — Probability; 


CA ESSA STUDY 7 


Students with weights of greater than 10 were dropped from analyses, as weights of these 
magnitudes are indicative of individual students who would have outsized influence on analytic 
results. This only occurred in a handful of observations and did not appreciably change the 
makeup of the comparison samples. 


The result of these PSW procedures was that comparison students who were more similar 
to treatment students (in terms of prior achievement and demographic covariates) were weighted 
more heavily in the analyses, and comparison students who were less similar to treatment 
students were weighted less. This approach resulted in the creation of weighted comparison 
groups at each grade level that were as similar as possible to the observed groups of treatment 
students. After these weights were applied to comparison students, baseline equivalence was 
achieved for fall 2020 mathematics scores across all grades of students, with standardized mean 
differences all having magnitudes of less than 0.24. These adjusted mean scores can be found by 
grade level and subject in Appendix A. However, after taking into account attrition, which was 
especially high in the upper grade levels, baseline equivalence remained in grades 3-6, but was 
not achieved in grades 7 and 8, even with the application of propensity score weights. As a 
result, these grade levels were not included in further analyses. 


Results 


i-Ready usage. We first descriptively examine patterns of i-Ready usage by school level. 
Usage data refers only to i-Ready Personalized Instructional usage time, and not to Diagnostic 
assessment time, so only treatment students had usage data. Grades 3-5 were grouped together as 
Elementary students, and grade 6 students were considered Middle students, as grades 7 and 8 
were excluded from analysis. Table 4 shows descriptive statistics relating to i-Ready math usage 
metrics. 


Table 4 
i-Ready usage means and standard deviations for elementary and middle school students in 
Mathematics 


Elementary (grades 3-5) Middle (grade 6) 
Total lessons 35.88 (26.49) 19.72 (19.49) 
Unique lessons 30.41 (21.58) 16.47 (15.69) 
Passed lessons 29.64 (22.41) 14.86 (14.88) 
Minutes of Usage 1077.27 (714.39) 824.49 (751.15) 
N 5427 822 


Usage metrics were consistently higher for elementary students than they were for middle 
school students, and to a considerable degree. Elementary students averaged approximately 16 
more total lessons, 14 more unique lessons, 15 more passed lessons, and slightly greater than 
four more hours of usage than did middle school students. 


Achievement descriptive statistics. We now examine descriptive patterns of 
achievement throughout the 2020-21 school year. We present fall 2020 i-Ready and spring 2021 
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MCAS scores, as MCAS scores were the main outcome variable in our analyses. These scores 
are displayed by grade in Table 5. 


Table 5 
Average i-Ready and MCAS mathematics scores, 2020-21 
Fall i-Ready Treatment Comparison Spring MCAS Treatment _ Comparison 


Grade 3 416.14 438.18 475.39 497.95 
Grade 4 434.50 452.71 472.84 494.15 
Grade 5 452.01 469.54 479.96 489.02 
Grade 6 465.02 482.63 476.58 490.33 


As noted previously, comparison students consistently scored higher on the fall i-Ready 
Diagnostic assessment than did treatment students. This difference may be related to 
characteristics of schools that decided to implement i-Ready Personalized Instruction as opposed 
to the Diagnostic-only program, but another important consideration is that, in the middle school 
grades, i-Ready Personalized Instruction is typically used for remediation purposes. Unadjusted 
spring MCAS scores also tended to be higher, on average, for comparison students, with 
differences ranging from 10-22 points. Further descriptive analyses may be found in Appendix 
B. 


Grade-level achievement analyses 


In this section, we present the results of grade-level analyses examining the effect of i- 
Ready Personalized Instruction, in relation to Diagnostic-only usage. We will present results for 
each of grades 3-6 in mathematics. Separate analyses were conducted on each grade-level, 
resulting in a total of 4 separate regression analyses. 


Grade-level analyses. Table 6 shows the results of grade-level analyses examining the 
effect of i-Ready Personalized Instruction on MCAS mathematics scores. Separate analyses were 
performed for each grade level. 


Table 6 

Grade-level analyses of i-Ready Personalized Instruction on MCAS Mathematics scores 
Grade Estimate Standard Error P-value Effect size 
Grade 3 (220° 2.928 .014 0.306 
Grade 4 3.550 5.773 ote) 0.170 
Grade 5 5.860*** 1.376 <.001 0.310 
Grade 6 5.145** 1.946 .008 0.275 


Note: * p <.05, ** p <.01, *** p <.001 


Statistically significant positive effects of i-Ready Personalized Instruction on MCAS 
mathematics scores were observed in grades 3, 5, and 6. I-Ready Personalized Instruction usage 
was associated with a greater than 7-point gain in MCAS mathematics scores for grade 3 
students, nearly a 6-point increase for grade 5 students, and just over a 5-point increase for grade 
6 students. Effect sizes were also robust in these three grades, indicating that treatment students 
scored nearly one-third of a standard deviation higher than did comparison students. Even in 


CA ESSA STUDY 9 


grade 4, with a non-significant finding, the directional association with i-Ready Personalized 
Instruction was positive. 


Grade-band analyses. We also conducted a series of supplementary analyses in which 
we examined the impact of i-Ready Personalized Instruction across grade bands. Specifically, we 
defined the “elementary” grade band as consisting of students in grades 3-5, while we defined 
the “middle” grade band as consisting of students in grade 6. The models used in these analyses 
are identical to those used in the grade-level analyses, with the addition of dummy variables to 
control for student grade level. The results of these analyses are shown in Table 7. 


Table 7 

Impacts of i-Ready Personalized Instruction on MCAS mathematics achievement, by grade band 
Grade Estimate Standard Error P-value Effect size 
Elementary Syie eo aneaae 1.174 <.001 0.278 
Middle 9.145** 1.946 .008 0.275 


Note: *** p <.001, ** p <.01 


Consistent with the grade-level analyses, results were positive across both grade bands. I- 
Ready mathematics Instruction had a statistically significant positive impact on MCAS scores for 
elementary students, with i-Ready Personalized Instruction students averaging nearly 6 points 
higher on the MCAS mathematics assessment than comparison students. Instruction impacts 
were also evidenced in the middle grades, as students who received i-Ready Personalized 
Instruction averaged approximately 5 points higher on the MCAS mathematic assessment than 
did comparison students. 


Combined analysis. As an additional set of supplementary analyses, we also conducted 
analyses using the entire sample of grades 3-6 students to examine the effect of i-Ready 
Personalized Instruction on mathematics achievement. These analyses were identical to the 
grade-level analyses and included the same propensity score weights. To account for grade-level 
differences, a set of dummy variables was included in these models to control for grade. Results 
of the overall analyses for math achievement are found in Table 8. 


Table 8 

Impacts of i-Ready Personalized Instruction on mathematics achievement, across grades 3-6 
Grade Estimate Standard Error P-value Effect size 
i-Ready 5.427*** 1.051 <.001 0.258 
Personalized 
Instruction 


Note: *** p <.001 


The estimated impact of i-Ready Personalized Instruction on MCAS mathematics scores 
was Statistically significant, with students who received i-Ready Personalized Instruction scoring 
nearly 5.5 points higher, on average, on the MCAS mathematics assessment than did comparison 
students. Taken together, the results of these analyses show that i-Ready Personalized Instruction 
had a statistically significant positive impact on mathematics achievement across all of grades 3- 
6. 
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Usage Analyses 


Next, we present a series of analyses examining the associations between i-Ready usage 
metrics and achievement. These analyses were identical to the previous achievement analyses, 
with the treatment variable being replaced with an i-Ready usage variable in each model. 
Separate models were run for each variable in each grade level. i-Ready usage metrics used in 
these analyses include total hours of usage and counts of completed lessons, unique lessons, and 
passed lessons. We follow this up with similar analyses examining the relationships between 
instructional usage quartiles and achievement, as well as the effects of meeting Curriculum 
Associates’ usage guidelines on achievement. 


Math usage. We present the results of analyses examining the effects of i-Ready 
Personalized Instruction variables on mathematics achievement. Table 9 shows the 
unstandardized coefficients of the same four usage variables as in the previous analyses, across 
each grade. 


Table 9 

Associations between i-Ready Personalized Instruction usage and mathematics achievement 
Grade Total time Completed Unique Lessons Passed Lessons 

(hours) Lessons 

Grade 3 0.470*** 0.224*** Q20are* 0.294%" 
Grade 4 03522" ™ O.183** 0.249*** O2a1t** 
Grade 5 0.264*** 0.141*** 0.20.17** 0.211+** 
Grade 6 0.240*** OS AS ata O.215"% 0.264*** 


Note: *** p <.001 


i-Ready usage metrics were significantly positively associated with MCAS mathematics 
scores. All of the coefficients listed in Table 9 were statistically significant at the .001 alpha 
level. Coefficients in Table 9 can be interpreted as the expected change in MCAS mathematics 
score for every unit of a usage variable. For example, each hour of i-Ready Personalized 
Instruction usage was associated with a nearly half-point MCAS mathematics score increase for 
grade 3 students. Similarly, each completed lesson for a grade 3 student was associated with a 
nearly quarter-point increase in MCAS math score, and each unique and passed lesson was 
associated with a nearly three-tenths of a point MCAS math score increase. The largest 
magnitudes of associations were generally found in the elementary grades. 


Usage quartiles. We next examined the relationships between i-Ready Personalized 
Instruction quartiles, as measured by hours of total usage, and achievement, as measured by 
MCAS scores. Usage quartiles were created because, in previous analyses relating curriculum 
usage and achievement, these relationships tended to be curvilinear, with the lowest and highest 
users tending to have the highest levels of achievement. Thus, these relationships are potentially 
underestimated in simple correlational and regression analyses. The construction of usage 
quartiles allows for a more partitioned investigation of the relationship between instructional 
usage and achievement for different levels of usage. We constructed usage quartiles on the basis 
of minutes of instructional usage for students with non-missing MCAS scores, with quartile 1 


CA ESSA STUDY il 


indicating the lowest levels (i.e., 25" percentile and lower) of usage and quartile 4 (i.e., 75" 
percentile and higher) indicating the highest levels of usage. This was done at each grade level. 
A positive association would indicate that a usage quartile was associated with higher MCAS 
scores, in relation to comparison students, who had no instructional usage. 


Analyses were performed for mathematics usage quartiles and achievement. The models 
used in these analyses are identical to those used in previous variables, with the treatment 
variable being replaced by four dummy variables representing the quartiles of usage. Tables of 
usage quartiles ranges are shown in Table 10. 


Table 10 
Usage quartile ranges (in hours) and sample sizes, by grade 
Grade Quartile1 N Quartile 2 N Quartile 3 N Quartile 4 N 


Grade 3 <9.94 483 9.94-17.74 484 17.75-26.02 484 >26.02 483 
Grade 4 <8.98 451 = 8.98-15.67, 453): 15.68-24.25 449 = >24.25 451 
Grade 5 <8.18 422 8.18-15.45 423 = 15.46-23.32) 421 = >23.32 423 
Grade 6 <4.20 232 4.20-10.30 = 233 = -10.31-18.53 234) >18.53 232 


Usage quartiles in elementary school were considerably larger than in middle school in 
terms of math usage, with median mathematics usage ranging between 15.46-17.75 hours for 
elementary students. By contrast, median mathematics usage in middle school was 10.31 hours. 


Usage quartiles and mathematics achievement. We now present the results of grade- 
level analyses examining the association between i-Ready Personalized Instruction usage 
quartiles and MCAS mathematics scores. Table 11 shows the unstandardized regression 
coefficients for each usage quartile, in relation to comparison students, for each grade. 


Table 11 
Associations between i-Ready Personalized Instruction usage quartiles and mathematics 
achievement 


Grade Quartile 1 Quartile 2 Quartile 3 Quartile 4 
Grade 3 5.987 1.0;795*** 15.883*** 21,945""* 
Grade 4 1.046 5.394 8.585 12,121* 

Grade 5 4.884*** 7.615*** 8.758*** 13.202*** 
Grade 6 3.938* 6.035** 8.400** 12.801*** 


Note: * p <.05, ** p <.01, *** p < .001 


Positive significant relationships between usage and mathematics achievement were 
predominant across most grades. Specifically, the strongest positive associations were observed 
in grades 3, 5, and 6, with all or nearly all quartiles of instructional usage associated with 
significantly higher MCAS mathematics scores in relation to comparison students. A statistically 
significant positive relationship between quartile 4 usage and mathematics achievement was also 
observed in grade 4. In general, quartile 4 usage was associated with the strongest effects on 
mathematics achievement, as evidenced by a statistically significant positive association in all 
grades, and quartile 1 usage was associated with the weakest effects, as statistically significant 
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positive effects were only observed in grades 5 and 6. Similarly, the magnitudes of effects were 
generally largest for quartile 4 usage and weakest for quartile 1 usage. 


Curriculum Associates usage guidelines. Curriculum Associates provides 
recommended i-Ready Personalized Instruction usage guidelines to educators. Specifically, 
Curriculum Associates recommends individual students aim for a consistent 30-45 minutes of i- 
Ready Personalized Instruction usage per subject per week and an average of at least 70% of 
lessons passed for the year. However, to identify students who met Curriculum Associates’ 
recommended guidelines, and consistent with previous i-Ready Personalized Instruction efficacy 
studies, we operationalized this guidance as follows: 


e At least 18 weeks of i-Ready Personalized Instruction use 
e An average of at least 30 minutes per week of Instruction use 
e An average lesson pass rate of greater than 70% 


Although at least 18 weeks of i-Ready Personalized Instruction use is not formal educator 
guidance, this rule was included to ensure consistent usage of i-Ready Personalized Instruction. 
Similarly, while educators are recommended that 45 minutes per week of Instruction usage is 
recommended, 30 minutes per week of Instruction usage has been a common usage benchmark 
for Curriculum Associates. Students that met all three of these guidelines were classified as 
having met usage guidelines, while those that did not meet all three criteria were classified as not 
having met usage guidelines. Table 12 displays the percentages of i-Ready Personalized 
Instruction students who met i-Ready Personalized Instructional usage guidelines. 


Table 12 

Percentages of students meeting i-Ready Mathematics Instruction usage guidelines 
Grade Met i-Ready usage guidelines 
Grade 3 (n = 1784) 62.33 
Grade 4 (n = 1666) 53.18 
Grade 5 (n= 1570) 44.71 
Grade 6 (n = 822) 28.83 


Over half of the i-Ready mathematics Instruction students met usage guidelines in grades 
3 and 4, while percentages of students reaching usage guidelines declined through later grades, 
with about 45% of grade 5 students and only 29% of grade 6 students reaching usage guidelines. 


We also conducted grade-level analyses that examined the relationship between meeting 
or not meeting i-Ready usage guidelines on achievement, in relation to comparison students. The 
models used in these analyses are similar to those used in previous analyses, except the treatment 
variable is replaced by dichotomous variables indicating i-Ready Personalized Instruction 
students who did or did not meet usage guidelines. Comparison students were assigned a “0” for 
both dichotomous variables. This allowed us to uniquely estimate the relationship between 
MCAS achievement and instructional usage that did or did not meet Curriculum Associates’ 
guidelines. Results of these analyses are shown in Table 13. 
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Table 13 

Associations between meeting i-Ready usage guidelines and MCAS mathematics achievement 
Grade Did not meet usage guidelines Met Usage Guidelines 
Grade 3 6.644* 16.66*** 
Grade 4 1.893 9.222 
Grade 5 5.030"** 10,39 7%** 
Grade 6 4.638** 12,.641*** 


Note: * p <.05, ** p <.01, *** p < .001 


Patterns of achievement gains differed between treatment students who did and did not 
meet i-Ready Personalized Instructional usage guidelines in mathematics, even though most of 
the associations were significant and positive. Any sort of i-Ready usage, whether it met 
recommended usage guidelines or not, was associated with significantly higher MCAS 
mathematics scores in grades 3, 5, and 6, although the magnitude of the increases was 
consistently lower for treatment students who did not meet usage guidelines. In these grades, 
treatment students who met usage guidelines averaged over 10-point larger MCAS mathematics 
gains than did comparison students, while treatment students who did not meet usage guidelines 
averaged 4.5-7-point MCAS mathematics gains. These results suggest the importance of students 
to not just receive i-Ready Personalized Instruction, but to meet instructional usage guidelines, to 
fully realize the potential benefits of i-Ready Personalized Instruction on achievement. 


Discussion 


The purpose of this evaluation was to examine the impact of i-Ready Personalized 
Instruction on mathematics achievement, as measured by MCAS scores. We compared students 
who received i-Ready Personalized Instruction with students who only participated in i-Ready 
Diagnostic testing. We also examined associations between various i-Ready usage metrics and 
achievement. 


In interpreting the findings of this evaluation, some limitations should be noted. First, 
while we controlled for as many demographic variables as possible, some, namely economic 
disadvantage and ELL status, were not available from all school districts involved in this 
evaluation, meaning we were unable to control for these variables or conduct relevant subgroup 
analyses. Similarly, we had access only to spring MCAS scores and i-Ready Diagnostic score 
and usage data from the 2020-21 school year. This limited our analyses to only one year and to 
strictly quantitative measures, which precluded drawing any conclusions regarding the fidelity of 
implementation within classrooms by teachers and students, outside of the quantitative usage 
data supplied to us by Curriculum Associates. Qualitative implementation data would have been 
instrumental in explaining differences in instructional usage between grades and schools, 
especially given the rapidly changing school environments of the 2020-21 school year caused by 
the COVID-19 pandemic. 


Grade-level Achievement Gains 


Statistically significant positive effects of i-Ready Personalized Instruction on MCAS 
mathematics scores were observed in grades 3, 5, and 6 throughout the 2020-21 school year. 
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Treatment students in these grades averaged 5-7 points higher on the MCAS mathematics 
assessment than did comparison students. Supplementary analyses showed that, when combined 
across grade bands, i-Ready Personalized Instruction had a statistically significant positive 
impact on elementary students’ mathematics achievement, as well as on grade 6 students’ 
mathematics achievement. It is important to note that the present analyses compared the 
incorporation of i-Ready’s Instruction component to the Diagnostic Assessment component only, 
which itself has already been shown to be an effective research-based intervention. Thus, it is 
likely that efficacy estimates of the combined Instruction with Diagnostic Assessment treatment 
condition may have been conservative, relative to a design where comparison students received 
neither of the i-Ready components. 


Usage Patterns 


Descriptive analysis of usage by grade revealed that usage metrics were generally greater 
in elementary school than in middle school. Elementary treatment students averaged 
approximately four more hours of mathematics usage than did middle school treatment students. 
Similarly, elementary treatment students averaged considerably more completed lessons, unique 
lessons, and passed lessons in mathematics i-Ready Personalized Instruction than did middle 
school treatment students. 


Regression analyses with usage variables showed that each of the four usage metrics we 
considered were significantly positively related to student mathematics achievement. These 
relationships were consistently strong and positive across all grade levels. 


When examining associations between usage quartiles and achievement, statistically 
significant associations between usage quartiles and MCAS mathematics achievement scores 
were observed. Nearly all quartiles of mathematics usage were consistently associated with 
significantly higher MCAS math scores in relation to comparison students. This pattern held 
across all grades, with higher quartiles (i.e., quartiles 3 and 4) of usage associated with the 
greatest MCAS mathematics score gains. Notably, larger proportions of students met i-Ready’s 
instructional usage guidelines in elementary school than in middle school. This may be attributed 
in part to i-Ready Personalized Instruction being used largely for remediation purposes in middle 
school, which is different from how i-Ready Personalized Instruction is typically implemented in 
elementary schools. Grade-level analyses showed that students who met usage guidelines 
showed significant achievement gains at nearly every grade level. In addition, students in grades 
3, 5, and 6 who used i-Ready Personalized Instruction but did not meet usage requirements still 
significantly outscored comparison students. Magnitudes of effects were consistently larger for 
students who met usage guidelines, in relation to those who did not meet guidelines. 


Conclusions 
The key results and conclusions of this evaluation are as follows: 
e Students in grades 3, 5, and 6 who used i-Ready Personalized Instruction showed 


significantly higher mathematics achievement on the MCAS mathematics assessment 
than did comparison students. 
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e i-Ready Personalized Instruction metrics including total time and lesson count variables 
were significantly positively associated with mathematics achievement across all grades. 

e i-Ready Personalized Instruction usage was generally higher in elementary grades, in 
relation to middle school grades. 

e All quartiles of i-Ready Personalized Instruction usage were associated with significantly 
higher mathematics achievement, in relation to comparison students, in grades 5 and 6. 
Quartiles 2-4 of i-Ready Personalized Instruction usage were associated with 
significantly higher mathematics achievement in grade 3, as well. 

e Students who met i-Ready Personalized Instruction usage guidelines generally showed 
significantly higher MCAS mathematics achievement, in relation to Diagnostic-only 
students. 


Appendix A: ESSA Tables 
Table Al 
Baseline equivalence by grade 
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Overall Treatment Control Adjusted T Pooled Stan. 

Mean Mean Mean VC Unadjusted Mean 

(SD) (SD) Difference SD Diff. 

3rd grade 420.28 416.14 425.31 -5.39 25.10 -0.215 
(26.42) (20.12) 

4th grade 437.11 434.50 439.94 -5.98 25.14 -0.238 
(26.23) (22.23) 

5th grade 452.98 452.01 453.97 2.77 27.08 0.102 
(27.69) (25.98) 

6" grade 467.53 465.02 470.42 4.97 29.18 0.170 
(31.89) (25.36) 

7™ srade 474.48 474.39 473.60 7.65 29.76 0.257 
(32.00) (27.21) 

8" srade 484.01 483.00 485.03 10.01 36.28 0.276 


(37.02) (35.03) 


NOTE: SD=standard deviation; all estimates include propensity-score weights. Baseline equivalence was calculated 
only for students with non-missing pretest and posttest data. 


Table A2 

Summary of student attrition, by grade 
C T N N Attrited Attrited T Overall Differential 
Student Student Randomized Randomized C Students Student Student 
N N to C to T Students Attrition Attrition 

Rate (%) Rate (%) 

540 1814 557 1934 17 120 5.50 3.15 
684 1690 703 1804 19 114 5.31 3.54 
912 1592 944 1689 32 97 4.90 2.35 
653 833 698 931 45 98 8.78 4.08 
705 706 746 785 41 79 7.84 4.56 


327 626 428 704 101 78 15.81 12.52 


CA ESSA STUDY 
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Table B1 
Sample sizes, by grade and district, 2020-21 
Treatment Comparison 

District 1 
Grade 3 58 276 
Grade 4 8 312 
Grade 5 21 343 
Grade 6 50 295 
Grade 7 7 346 
Grade 8 0 0 
District 2 
Grade 3 110 0 
Grade 4 0 103 
Grade 5 5 100 
Grade 6 0 103 
Grade 7 0 90 
Grade 8 0 88 
District 3 
Grade 3 19 64 
Grade 4 4 50 
Grade 5 12 241 
Grade 6 0 20 
Grade 7 0 4 
Grade 8 0 7 
District 4 
Grade 3 0 197 
Grade 4 0 214 
Grade 5 2 220 
Grade 6 7 217 
Grade 7 1 217 
Grade 8 3 170 
District 5 
Grade 3 1597 2 
Grade 4 1654 5 
Grade 5 1530 8 
Grade 6 765 18 
Grade 7 671 46 
Grade 8 599 61 

Table B2 


Sample sizes by grade, 2020-21 
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Treatment Comparison 

Grade 3 1784 539 
Grade 4 1666 684 
Grade 5 1570 912 
Grade 6 653 822 
Grade 7 703 679 
Grade 8 326 602 

Table B3 

Average math achievement scores, by grade and district, 2020-21 

Treatment Comparison Treatment Comparison 

District 1 Fall i-Ready Spring MCAS 
Grade 3 424.84 439.63 502.28 498.96 
Grade 4 n/a 454.10 n/a 498.33 
Grade 5 465.62 474.13 490.86 489.00 
Grade 6 490.90 482.29 494.36 489.16 
Grade 7 n/a 497.60 n/a 495.47 
Grade 8 n/a n/a n/a n/a 
District 2 Fall i-Ready Spring MCAS 
Grade 3 425.38 n/a 479.75 n/a 
Grade 4 n/a 443.78 n/a 474.39 
Grade 5 n/a 455.79 n/a 475.34 
Grade 6 n/a 472.97 n/a 483.81 
Grade 7 n/a 486.53 n/a 480.41 
Grade 8 n/a 509.84 n/a 489.77 
District 3 Fall i-Ready Spring MCAS 
Grade 3 415.37 432.77 476.47 484.66 
Grade 4 n/a 454.30 n/a 495.76 
Grade 5 456.67 471.32 488.42 491.84 
Grade 6 n/a 486.20 n/a 489.75 
Grade 7 n/a n/a n/a n/a 
Grade 8 n/a n/a n/a n/a 
District 4 Fall i-Ready Spring MCAS 
Grade 3 n/a 438.02 n/a 501.19 
Grade 4 n/a 454.87 n/a 497.93 
Grade 5 n/a 473.23 n/a 492.87 
Grade 6 n/a 488.59 n/a 496.69 
Grade 7 n/a 501.94 n/a 495.65 
Grade 8 n/a 516.11 n/a 492.73 
District 5 Fall i-Ready Spring MCAS 
Grade 3 415.19 n/a 474.10 n/a 
Grade 4 434.46 n/a 472.76 n/a 
Grade 5 451.84 n/a 479.76 n/a 
Grade 6 463.56 467.56 475.50 470.94 
Grade 7 474.71 473.61 477.57 477.09 
Grade 8 483.07 484.90 475.79 473.97 
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i-Ready usage means and standard deviations for the 2020-21 school year, by grade 
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Grade 3 


N = 1784 


Total lessons 
Unique lessons 
Passed lessons 
Minutes of Usage 


41.21 (27.55) 
35.91 (23.27) 
35.59 (24.00) 
1171.96 (743.64) 


Grade 4 


N = 1666 


Total lessons 
Unique lessons 
Passed lessons 
Minutes of Usage 


36.21 (25.74) 
30.62 (21.03) 
29.78 (21.80) 
1087.35 (693.50) 


Grade 5 


N = 1550 


Total lessons 
Unique lessons 
Passed lessons 
Minutes of Usage 


30.14 (22.65) 
24.74 (17.35) 
23.35 (17.78) 
1023.36 (675.06) 


Grade 6 


N = 822 


Total lessons 
Unique lessons 
Passed lessons 
Minutes of Usage 


41.21 (27.55) 
35.91 (23.27) 
35.59 (24.00) 
1171.96 (743.64) 


Grade 7 


N= 679 


Total lessons 
Unique lessons 
Passed lessons 
Minutes of Usage 


16.62 (15.93) 
13.72 (12.86) 
11.88 (12.27) 
832.71 (799.45) 


Grade 8 


N = 602 


Total lessons 
Unique lessons 
Passed lessons 
Minutes of Usage 


13.93 (14.08) 
11.16 (10.91) 
9.52 (10.31) 
703.36 (673.49) 


Table B5 
i-Ready usage means and standard deviations for the 2020-21 school year, by district 
District 1 N= 144 
Total lessons 3.19 (3.83) 
Unique lessons 3.08 (3.60) 
Passed lessons 2.91 (3.60) 
Minutes of Usage 59.43 (67.37) 
District 2 N= 115 


Total lessons 
Unique lessons 
Passed lessons 


34.02 (25.86) 
30.29 (22.78) 
30.80 (23.77) 
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Minutes of Usage 


893.17 (806.45) 
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District 3 


N=35 


Total lessons 
Unique lessons 
Passed lessons 
Minutes of Usage 


15.66 (14.80) 
14.17 (13.58) 
13.63 (13.44) 
325.24 (303.64) 


District 4 N= 13 

Total lessons 7.00 (7.47) 
Unique lessons 6.46 (6.78) 
Passed lessons 6.23 (6.78) 
Minutes of Usage 127.76 (155.53) 
District 5 N = 6816 


Total lessons 
Unique lessons 
Passed lessons 
Minutes of Usage 


31.10 (25.13) 
26.26 (20.73) 
25.13 (21.31) 
1034.59 (9.62) 


Appendix C: Grade-level Regression Tables 


Table C1 
Overall impact of i-Ready Personalized Instruction on spring 2021 mathematics achievement, by 
grade 
Variable Estimate Standard Error | P-value Effect Size 
Grade 3 
i-Ready Personalized Instruction 722" 2.923 .014 0.306 
Constant 473.57*** 2.53 <.001 
Variance of constant 29132 
Residual 315.92 
Student NV 2126 
Class N 44 
Grade 4 
i-Ready Personalized Instruction 3.59 Bee 009 0.170 
Constant 474.94*** 4.69 <.001 
Variance of constant 8.02 
Residual 208.05 
Student NV 2136 
Class N 43 
Grade 5 
i-Ready Personalized Instruction 5.860*** 1.38 <.001 0.310 
Constant 478.95*** 1301 <.001 
Variance of constant 6.00 
Residual 7 Lod 
Student NV 2260 
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Class N 43 
Grade 6 
i-Ready Personalized Instruction plot" 1.95 .008 0.275 
Constant ATT. 7 ar ee 1.90 <.001 
Variance of constant 6.84 
Residual 168.05 
Student N 1251 
Class N 16 


